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Overview 


SSIP Phase | plans require states to determine a “state- 
identified measurable result” (SiMR) through a systematic 
process that is summarized graphically through a 

Theory of Action (ToA). A TOA sets forth assumptions or 
hypotheses about how an improvement strategy works. 
These hypotheses are then tested during an evaluation. The 
power of a TOA is in its specificity of thought and in the 
explicit reasoning that calls attention to the essential steps 
and checkpoints in implementing an improvement strategy. 
Thus, the ToA establishes a clear path toward the end goal of 
affecting outcomes for children and youth with disabilities 
and their families. 


Importantly, the ToA acts as a set of checkpoints to evaluate if 
the expected outcomes at each step along the way are 
realized and, if they are not, suggests where to intervene to 
get the improvement strategy back on track. 


This paper demonstrates how the work completed under SSIP 
Phase | contributes to a key component of the activities required 
during Phase Il: an evaluation plan. By using the ToA as an 
outline or a roadmap for the evaluation plan, state SSIP leaders 
can identify and measure, through performance indicators, the 
checkpoints necessary for marking progress toward the SiMR. 
The first two sections of this white paper illustrate the 
relationship between SSIP Phase | and Phase II and briefly 
demonstrate how the ToA can be used to develop the 
evaluation plan and formulate evaluation questions. The third 
section shows in detail how SSIP leaders can use a TOA to 
develop performance indicators that measure progress toward 
the SiMR. 


The Relationship Between SSIP Phase I| and Phase II 


Planning an evaluation and conducting an evaluation are 
companion processes. Unfortunately, different audiences use 
different terms for similar concepts. The resulting confusion 
undermines integration of planning and evaluation, so it is 
important to make the connection between the different 
terms. As illustrated in Exhibit 1, the SSIP evaluation plan 
(Phase Il), directly proceeds from the activities conducted in 
Phase |; that is, from 


e  Ananalysis of student outcomes, including drilling deep 
into the data to understand the performance of 
disaggregated student groups (e.g., ages, disability, type of 
outcome, location); 
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e  Ananalysis of infrastructure strengths and weaknesses, 
including governance, fiscal, technical assistance, and 
monitoring systems; 


e Discussions by stakeholder groups, who thoughtfully 
frame an end result for the program’s efforts, called a SiMR; 
and 


e Identification of coherent improvement strategies that 
will build state and local education agency (LEA) capacity 
to achieve the SiMR. 
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Exhibit 1. Linking SSIP Phase | to SSIP Phase II 


THEORY Infrastructure Data 
OF ACTION Analysis Analysis 


EVALUATION Research raaeeton wee 
PLAN uestion 
Q Outcomes (SIMR) 
The state’s response to improving the SiMR is articulated Evaluation plans are strengthened by collecting evidence of 
graphically in a ToA and can be specified as strategies, actions, impact at several levels or steps in an outcomes hierarchy; 
tactics, or a host of other terms. The strategies generate information on outcomes from the proximal steps helps to 
potential evaluation questions, provide insights on the explain outcomes at the distal steps, which take longer to 
program’s activities and some or all of the targeted audiences, achieve. All steps in the hierarchy are necessary outcomes and 
make explicit short-term and intermediate outcomes, and demonstrate progress toward the results-oriented outcome, the 
clarify the long-term results desired by the program. SIMR (see Exhibit 2). 
Exhibit 2. Hiera rchy of possible To illustrate, a program’s initial intended aim is to obtain 
e 
participation among targeted staff and positively influence their 

outcomes reactions to the program activities that affect their learning 


results for children, youth, and their families organization implementing the program with fidelity in the 
relevant setting and children and youth with disabilities and 


their families demonstrating continued progress toward 
expected outcomes within the established timeframe. 
Concurrently, systems and environmental changes in conditions 
are the result of recommendations, actions, policies, and 
practices. 


ACTIONS 
New skills used and patterns of behavior 
adopted by target audiences 


LEARNING 
New knowledge and skills as end results 


REACTIONS 

Readiness; degree of interest toward the 
program; acceptance of activities and of 
education methods 


PARTICIPATION 

Number of people reached; characteristics 
of the frequency, intensity, and quality of 
contact 


'For guidance considering and assessing the policies, procedures, 
processes, and practices needed for a high-quality Part B data 
system, see the /DEA Data Center Part B Data System Framework, 
available at 
https://www.ideadata.org/resources/resource/1593/idea-data- 
center-part-b-data-system-framework. 


Source: Adapted from Centers for Disease 
_ ee and Prevention (1999). 
<< 


SYSTEMS AND ENVIRONMENT CHANGE keg knowledge, Opimonseskills) 

Through this learning, people and organizations take actions 
CHILD, YOUTH, AND FAMILY OUTCOMES that result in behavioral and social change. This includes the 
Learning, social-emotional, and behavioral eee ‘ fit Gat ae 
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Using a Theory of Action to Develop an SSIP Evaluation Plan 


When developing the SSIP evaluation plan, SSIP state 
leaders need not start from scratch. Instead, the ToA 
provides an initial direction for the evaluation plan. 


Exhibit 3 illustrates how a TOA for a multi-tiered system of 
supports (MTSS) strategy conveys three critical features of an 
evaluation plan: outcomes, target audiences, and activities. 


Exhibit 3. Theory of action as an outline for an evaluation plan 


coaches to elementary 
schools in target districts 
to assist in establishing 
robust multi-tiered 
systems of support... 


Then the LEA 


Key strands (e.g., teachersand Then teachers/ support 

of action If the SEA administrators) teams So that 
Multi-tiered | ...provides professional | ...will increase its capacity| ... monitor students with Reading 
systemsof | development to implement MTSS disabilities more closely in achievement for 
support opportunities and practices with fidelity... terms of their academic and | students with 


disabilities in 
elementary grades 
in target districts 
will improve. 


behavioral progress and 
needs 


...and more quickly and 
effectively provide supports 
and interventions that help 
students with disabilities be 
successful academically and 
behaviorally.... 


Note: SIMR Outcome in BOLD; targeted audiences in Italics; and activities are underlined; SEA = state education agency. 


Moving from right to left in the ToA above, the SIMR outcome 
defines what needs to be achieved and is an ambitious and 
longer term outcome: 


Increase the percentage of students with disabilities in schools 
in target districts that score at least Basic on the state 
achievement test for grades 3-5 in reading. The number of 
students with disabilities scoring at least Basic will increase by 
3 percentage points per year from the baseline score 
percentage. This rate of improvement constitutes an 
ambitious yet achievable goal that will ultimately raise the 
percentage of students with disabilities scoring Basic or higher 
by 15 percentage points over 5 years. 


There are targeted audiences that the program needs to urge 
into action in order to make progress on addressing these 
educational challenges. Generally, district and 
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school staff need to take action. Some action may be needed by 
the families and communities. Some actions may be necessary 
by constituencies all across the state infrastructure. Finally, 
teachers and service providers must instruct, engage, and 
progress-monitor students in the program to produce the 
favorable student-level outcomes (the SiMR). In the ToA above, 
target audiences are at the state, LEA, school, and classroom 
levels. 


It is the agency's role to specify the strategies or activities 
required to be in place to achieve the outcomes in the targeted 
groups. Activities will vary with the audience and program. In 
the ToA above, the coherent improvement strategy is a robust 
system of multi-tiered supports. The specific activities related to 
the strategy may include providing professional development 
to elementary schools in target districts (SEA- and LEA-level 


IDC 


Using a Theory of Action to Develop Performance Indicators to Measure Progress Toward a SiMR 


activity), providing coaches in schools in target districts (SEA- 
and LEA-level activity), establishing robust systems of multi- 
tiered supports in schools (LEA- and school-level 


activity), monitoring students and providing supports 
(classroom-level activity). 


Using a Theory of Action to Develop Evaluation 


Questions 


To be effective, an evaluation plan needs focus. This is where 
evaluation questions come in. There is no specific formula for 
writing evaluation questions; however, they must be asked in 

a manner that requires a response that can be accurately 3. 
measured, analyzed, and reported.2 Given that in many 

instances resources available for evaluations are limited, what 

do SSIP leaders and their stakeholders really need to know to 


understand how well the program’s activities are being 4. 
implemented and if the program has the expected impact on 

the target audiences and ultimately the SiMR population? 

Because the program activities and desired outcomes are é 


made transparent in the TOA, it is easy to use it to develop 
evaluation questions. 


Continuing with the example above, by aligning the 
evaluation questions to the hierarchy of outcomes, possible 
evaluation questions of the installation of MTSS practices 
include: 


1. Participation outcome. What proportion of teachers 
who were identified for receipt of the MTSS training 
participated? What proportion of the intended hours of 
professional development did teachers receive? What 
proportion of the intended hours of coaching did 
teachers receive? 


2. Reaction outcome. To what extent do the teachers 
agree that the MTSS practices can easily be 
implemented into theirclassroom instruction? Towhat 
extent do teachers report their readiness for using the 


MTSS practices? in their classroom and school? To what 
extent do teachers report the practices will improve 
student outcomes? 


Learning outcome. How well do teachers report that 
they understand how the MTSS practices influence 
student outcomes? To what extent do teachers rate their 
effective use of the MTSS practice highly? 


Action outcome. To what extent do teachers implement 
the MTSS practices with fidelity? Towhat extent do the 
teachers instruct students with the intended dosage and 
frequency? 


Child, youth, and familyoutcome(s): 


a. What is the impact of MTSS on the reading 
achievement of students with disabilities in schools 
in target districts compared to the reading 
achievement of students with disabilities in 
comparison schools? 


b. What is the impact of MTSS on the classroom 
behavior of students with disabilities in schools in 
target districts compared to the behavior of students 
with disabilities in comparison schools? 


c. What is the impact of MTSS on the reading 
achievement of all students in targetschools 
compared to the reading achievement of all students 
in comparison schools? 


? In these evaluation questions, MTSS practices represent the 


program to influence student outcomes. The data collection 
instruments, however, would include questions that address 
each MTSS practice. 
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3 In these evaluation questions, MTSS practices represent the program 
to influence student outcomes. The data collection instruments, 
however, would include questions that address each MTSS practice. 
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Identifying Performance!Indicators of Progress for the 


Evaluation Plan 


Identifying performance indicators helps guide and monitor 
progress toward the long-term outcome, or in the case of SSIP, 
the SiMR, and is important for several reasons: 


(1) When a new or large initiative is getting started, there may 
be clarity about the “big change” the program is to produce, 
but little else. (2) It is easy to place too much emphasis on 
listing program activities, many of which either cannot be 
measured or the measurement data necessary would 
overwhelm the evaluation and become burdensome to collect 
and costly to use. (3) Because the components of educational 
programs are often expressed in global or abstract terms, 
indicators—specific, observable, and measurable statements— 
help define exactly what we mean or are looking for and 
require the use of such terms as number of, percent of, mean 
of, or similar phrases. By working backward from the end 
outcomes and asking “how to,”SSIP leaders can identify what 
will be involved in producing change. 


Indicators provide clear definitions of global outcome 
statements such as “Teachers measure student reading 
progress twice a week” or “Families adopt in-home literacy 
techniques.” The teacher-based indicator might specify the 
type of reading intervention, the duration, or the adherence to 
the intervention’s fidelity, for example: 90 percent of teachers 
will measure oral reading fluency of students in Tier 3 once a 
week. Likewise, the family indicator might specify the in- 
home techniques and the intensity or duration of their 
adoption, such as, all families will read to their children at 
least five times a week for 20 minutes each time. Indicators 
such as these provide clearer definitions of the global 
statement and help SSIP leaders understand how to measure 
progress toward the end outcome. 


The use of indicators originates from a type of evaluation 
called performance measurement and can have many 
meanings. For our purpose, it is defined as regular 
measurement of the implemented service delivery and results 
(outcomes). Regular measurement of progress toward 
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specified outcomes is a vital component of any effort at 
managing for results. Performance measurement is a 
stakeholder-oriented process that focuses on maximizing 
benefit and minimizing negative consequences for the 
participants (Hatry, 2006). Specifically, a performance indicator 
is: an observable measure of both the process and theoutcome, 
at the child, classroom, teacher, family, school, or district level, 
which identities a specific numerical measurement indicating 
progress toward the outcome. As noted earlier, a performance 
indicator usually begins with words, such as number of, percent 
of, ratio of, proportion of, or mean, or another similar phrase. 


There are three levels or types of performance indicators, and 
each one contributes to marking progress toward the SiMR. 
Each one is defined below. 


Types of Performance Indicators* 


e Indicators of short-term outcomes: These outcomes 
are expected to lead to the desired end but are not 
themselves ends. There may be multiple short-term 
outcomes. Linking to the hierarchy of outcomes in 
Exhibit 2,examples of indictors of short-term outcomes 
include Participation outcome—percentage of teachers 
who rate highly the usefulness and feasibility of the 
program; Reaction outcome—percentage of teachers 
who express a high interest toward the program; 
Learning outcome—percentage of teachers who self- 
rate their positive changes in knowledge about the 
practice. 


e Indicators of intermediate outcomes: Again, linkingthe 
intermediate indicatorstothe hierarchy of outcomes in 
Exhibit 2, action outcome examples include, 
demonstrating the new practice or skill, 


4 Outputs from activities are not explicitly stated in these descriptions 
(e.g., number of courses offered, number or percentage of staff 
completing course). Managers and evaluators may elect to identify 
the outputs as part of a logic model sequence, but in this paper, the 
emphasis is on the indicators required for marking progress toward 
the SiMR (changes in behavior based on skills and knowledge). 
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including measures of the dosage, frequency, or quality of on child, youth, and family outcomes such as learning, social- 
services,? and are direct results of the improvement activities. emotional, behavioral child-, youth-, and family-level end 
Examples include frequency of MTSS meetings by teachers results. The indicator will specify to what extent students or 
and administrators or percentage of educators observed families will increase their academic or functional performance, 
implementing MTSS practices (e.g., continually progress- by how much, and over what time period. 


monitor students’ behavior and academic progress, provide 


supports across the tiers). 


5 


Linking the outcomes and performance indicators is essential to 
marking or evaluating progress toward the intended results set 
Indicators of long-term outcomes: These are the end forth by SSIP leaders. In the detailed example below, a state has 
results sought. For SSIP, this end result focuses identified that a subset of districts have large literacy 
achievement gaps between students with disabilities and their 


In the implementation science literature, these concepts are often peers in the early grades (see Exhibit 4). 


referred to as effort, quality, and fidelity (Fixsen, Naoom, Blase, 
Friedman, & Wallace, 2005). 


Exhibit 4. A hypothetical state rationale of a theory of action 
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SiMR:Improvereading achievementforelementary-age students with disabilities ina subset of districts with large 
achievement gaps between students with disabilities and their peers without disabilities. 


Data and root cause analysis: Our data analysis revealed that the achievement of students with disabilities in 
reading (as measured by the statewide assessment) in the elementary grades was significantly below that of their 
peers without disabilities. The achievement gap, in terms of the percentage of students proficient in reading, was 
highest in grades 3, 4, and 5. While there were variations and differences in achievement by students’ disability 
categories, race/ethnicity, and socioeconomic status, our stakeholder group (which included representation from 
local special education directors, special education teachers, general education teachers, school administrators, 
family advocacy groups, and higher education) was compelled by the overall level oflowachievementinthe 
elementary grades. They also noted that certain districts in the state had greater achievement gaps than others. 
Therefore, our state chose to focus its SIMR onimproving reading achievement in the elementary grades in a subset 
of districts with the largest achievement gaps. 


The state’s root cause analysis revealed a strong relationship between students’ educational environments and their 
academic achievement as measured by the statewide assessment. Specifically, elementary-age students educated 
in substantially separate placements as well as inside the regular class less than 40 percent of the day scored lower 
than similar students with disabilities educated in the regular classroom more than 80 percent of the day. In 
addition, a statewide survey of MTSS implementation revealed that districts reporting full implementation of an 
MTSS system also had higher rates of students with disabilities scoring proficient on the statewide assessment in 
reading than those reporting lower levels of implementation. Finally, our stakeholder group felt that poor 
instructional quality and lack of opportunities for general and special educators to co-teach were factors in the low 
achievement of some students with disabilities. 
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Based on the results of its root cause analysis, the state has 
identified a multi-pronged approach to improve academic 
outcomes for students with disabilities in reading in 
elementary schools in target school districts. This multi- 
pronged approach includes establishing robust MTSS in the 
elementary schools in target districts, 
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increasing access to the general education curriculum for 
students with disabilities, improving instructional quality, and 
encouraging co-teaching models (Exhibit 5). For the purposes 
of illustration, short-term, intermediate, and long-term 
outcomes and indicators are specified for just one of these 
strategies—MTSS (see Exhibit 6). 
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Exhibit 5. Example of a multi-pronged theory of action 


Illustrative example of a theory of action 


environments 


Instructional 
quality 


development 
opportunities and 
coaches to elementary 
schools in target districts 
to explore data on 
educational environment 
and analyze where 
students with disabilities 
are receiving their 
instruction... 


...provides professional 
development 
opportunities and 
coaches to elementary 
schools in target districts 
on research-based 
instructional strategies in 
literacy... 


informed and intentional 
decisions about 
educational placements 
for students with 
disabilities... 


..Will implement 
evidence- based practices 
with fidelity to meet the 
individual needs of 
students with 
disabilities... 


Key strands Then the LEA (teachers Then teachers and 
of action If the SEA and administrators) support staff 
Multi-tiered | ...provides professional ...Will increase its capacity | ...will monitor students with 
systems of development to implement MTSS disabilities more closely in 
support opportunities and practices with fidelity... terms of their academic and 
coaches to elementary behavioral progress and 
schools in target districts needs 
ee establishing ...and will more quickly and 
robust multi-tiered : ; 
systemnsor supper! effectively provide supports 
y and interventions that help 
students with disabilities be 
successful academically and 
behaviorally... 
Educational | ...provides professional ...will make more ...Will provide students with 


disabilities access to the 
general curriculum.... 


...Will deliver high-quality 


instruction.... 


Co-teaching 


...provides professional 
development 
opportunities and 
coaches to elementary 
schools in target districts 


to establish co-teaching 
models... 


...will offer general and 
special educators the 
opportunity to work more 
closely together in the 
classroom to design 
lessons, Support students, 
and braid best 
instructional strategies for 
students with disabilities 
and their peers... 


...will benefit from the 
shared knowledge and 


expertise of their colleagues 


in general and special 
education...and will 


provide students with 
disabilities access to the 
general curriculum.... 


So that 


Reading 
achievement for 
students with 
disabilities in 
elementary grades 
in target districts 
will improve. 


www.ideadata.org 


Using a Theory of Action to Develop Performance Indicators to Measure Progress Toward a SiMR 


IDC 


Exhibit 6. Illustrative example of a (partial) theory of action linked with 
performance indicators 


Short-term Outcomes 


Intermediate Outcomes 


Key strands Then the LEA (teachers Then teachers and 
of action If the SEA and administrators) support staff So that 
Multi-tiered ...provides professional ...Will increase its capacity ...will monitor students with Reading 
system of development opportunities | to implement MTSS disabilities more closely in achievement for 
supports and coaches to elementary | practices with fidelity... terms of their academic and students with 
schools in target districts to behavioral progress andneeds | disabilities in 
assist in establishing robust ...and will more quickly and elementary grades 
multi-tiered systems of effectively provide supports in target districts will 
support... and interventions that help Um 
students with disabilities be 
successful academically and 
behaviorally... 
Indicators of Indicators of Indicators of 


Long-term Outcomes 


Participation Outcomes: 


Percentage of participants who 
rated professional development 
opportunities as high quality, 
relevant, and useful 

Number of additional coaches 
hired 


Frequency of visits by coaches to 
elementary schools in target 
districts 

Percentage of participants who 
rated coaching sessions as high 
quality, relevant, and useful 


Reaction Outcome: 


Percentage of teachers reporting 
an interestin learning the program 


Learning Outcomes: 


Percentage of participants who 
reported increased knowledge 
abouteffective MTSS systems and 
practices 

Percentage of educators who 
reported increases in their 
application of knowledge and 
skills obtained through 
professional development 


Action Outcomes: 


Percentage of school sites at the installation stage of 


MTSS system and practices 


Percentage of school sites that developed MTSS 


systems frameworks 


Frequency of MTSS meetings by teachers and 
administrators convened at least once a month to e 
review data profiles of classes and students 


Percentage of educators who self-reported 

implementing MTSS practices (e.g., continually 
progress-monitor students’ behavior and academic e 
progress, provide supports across Tier I, Tier Il, and Tier 


III) 


Percentage of educators observed implementing MTSS 
practices (e.g., continually progress-monitor students’ 
behavior and academic progress, provide supports 


across the Tiers) 


Percentage of school sites demonstrating evidence of 
robust MTSSsystemsin elementary schools in target 
districts, including existence of high-quality Tier Il and 


Tier Ill 


Decrease in the percentage of students receiving Tier Ill 


and Tier Il supports 


Changes in the numbers and percentage of students 
receiving supports at each level over aset period of time 
as a proxy for demonstrating use of MTSS data-based 


management practices 


e Decrease in number of 
disciplinary referrals 


e Improvements in the 
academic growth in 
reading of students with 
disabilities 
Improvements in the 
academic growth of any 
students receiving Tier Il 
and Tier Ill supports 


Decrease in number of 
students referred or 
identified for special 
education 
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Summary 


This paper demonstrates how the activities completed under 
SSIP Phase | contribute to the evaluation planning 
component of SSIP Phase II, which then guides the evaluation 
work conducted in subsequent years. The ToA, as the product 
of data and infrastructure analysis, stakeholder discussions, 
and SiMR identification, visually depicts the underlying 
reasoning behind the choice of improvement strategies 
adopted by a state to achieve its long-term outcome (SiMR). 
Moreover, the ToA clearly defines the valued outcome(s), 
target audiences, and activities that a state will engage in 
over the life of its SSIP. Through this graphical argument, SSIP 
leaders and their stakeholders can generate important 
evaluation questions at each level of the outcomes hierarchy, 
identify essential steps and checkpoints in implementing 
selected improvement strategies, and evaluate if the 
expected outcomes at each step along the way are realized. 


SSIP evaluation plans are strengthened by periodic collection 
of evidence of impact at each level in an outcomes hierarchy. 
Regular measurement of progress toward specified outcomes 
is a vital component of any effort at managing for result as it 
assures SSIP leaders that the system is on track or indicates 
that mid-course changes may be necessary. Performance 
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indicators are acommon method of collecting this evidence as 
they are written in specific, observable, and measurable 
language to define exactly what SSIP leaders mean or are 
looking for and require the use of such terms as number of, 
percent of, mean of, or similar phrases. This paper describes 
three types of performance indicators. 


Indicators of short-term outcomes measure participation, 
reaction, and learning outcomes in the hierarchy of outcomes. 
Indicators of intermediate outcomes measure action outcomes 
in the hierarchy of outcomes. Finally, indicators of long-term 
outcomes measure the end results sought, which for the SSIP 
focuses on child-, youth-, and family outcomes such as learning, 
social-emotional, behavioral child-, youth-, and family-level end 
results. 


Woven throughout this paper is a practical illustration of how to 
use a TOA to develop evaluation questions and performance 
indicators linked to an outcomes hierarchy. While the example 
in the paper is focused on a single component of a multi- 
pronged TOA, the process outlined can be replicated to guide 
SSIP leaders through some of the steps necessary to develop 
the evaluation component of SSIP Phase Il. 
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